gh-153568: Pause the GC while converting the AST to Python objects - #153583
gh-153568: Pause the GC while converting the AST to Python objects#153583pablogsal wants to merge 1 commit into
Conversation
Freshly converted AST nodes cannot be part of a reference cycle yet, so collections during the conversion only pay to traverse the half-built tree without ever freeing any of it.
| // so a garbage collection while building them cannot free anything of | ||
| // this tree and only pays to traverse its half-built nodes. Pause the | ||
| // collector while the conversion runs. | ||
| int gc_was_enabled = PyGC_Disable(); |
There was a problem hiding this comment.
My, probably naive, question here is what about free-threading?
There seems to be a single global (per interpret) state:
cpython/Python/gc_free_threading.c
Lines 2527 to 2539 in 9d231cb
cpython/Python/gc_free_threading.c
Lines 1656 to 1661 in 9d231cb
For example: a thread disabled GC (in the window between here) but the parser would enable it again?
There was a problem hiding this comment.
Yes this will always be racy because is currently not possible to know if this was us or someone else doing it. This is something we need to consider when reviewing this optimization
There was a problem hiding this comment.
Maybe this is a more general problem?
Quick ripgrep reveals:
cpython/Modules/_posixsubprocess.c
Lines 1063 to 1067 in 1b9fe5c
cpython/Modules/_posixsubprocess.c
Lines 1326 to 1329 in 1b9fe5c
In reverse, maybe even more places would gain from disabling changing the default GC behaviour (disabling it?).
2 participants
Building the Python object tree out of the C AST allocates thousands of container objects in a burst, which repeatedly trips the garbage collector into traversing the half-built tree. Those nodes cannot be part of a reference cycle until the conversion exposes them, so a collection during the conversion can never free anything of this tree. Pausing the collector around the conversion (restoring the previous state, so a user-disabled GC stays disabled) removes those wasted traversals.
Benchmark (running
ast.parseover 8 of the largest stdlib files, 1.3 MB, 20 times per run; pinned cores):Executed instructions (stable under machine load,
perf stat) drop by 5.6%.